Measuring Text Reuse in a Journalistic Domain

This paper describes a general framework for measuring text reuse. This term is used to describe how content from a single or multiple number of known sources can be reused either verbatim (word-for-word copy) or otherwise rewritten depending upon factors influencing the creation of a new document. These may include reduction/ increase in length, change of style, simplification of content, shift in audience focus or disguise of original source. Text from a document that can, with confidence, be shown to have been rewritten from a source or sources is called derived. Measuring accurately the amount of reused text between documents would find applications in automatic plagiarism detection, detecting breaches of intellectual copyright and automatic evaluation of machine translation software. One area with such a need is that of reuse monitoring and control in journalism. A major UK newswire, the Press Association, would like to know how much of the text it produces is reused by the British press. A simple classifier using word overlap is tested that identifies 75% wholly-derived, 53% partiallyderived and 71% non-derived newspaper articles correctly.